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Abstract 


The Study of Deeper Learning: Opportunities and Outcomes-funded by the William and Flora Hewlett Foundation-aimed to determine 
whether students attending high schools with a mature and at least moderately well implemented approach to promoting deeper 
learning actually experienced greater deeper learning opportunities and outcomes than they would have had they not attended these 
schools. In this report-our third in a series of three-we focus specifically on key questions about student outcomes: Did students who 
attended participating network high schools perform better on tests of cognitive competency, report higher levels of interpersonal and 
intrapersonal competencies, or attain higher rates of high school graduation and college enrollment than they would have had they not 
attended the network schools? 

The analyses in this report focus on students from between 10 and 13 pairs of matched deeper learning network and comparison 
schools in California and New York (the number of pairs included in specific analyses varied based on the availability of data). After 
statistically accounting for differences in student background characteristics, we found that students who attended participating network 
high schools that explicitly focused on deeper learning experienced superior outcomes compared to students who attended non-network 
comparison high schools. Key takeaways include the following: 

1. On average, students who attended the network schools in the study achieved higher scores on the OECD PISA-Based Test for 
Schools (PBTS)-a test that assesses core content knowledge and complex problem-solving skills— than did similar students 
who attended non-network high schools. Students who attended network schools scored higher on all three PBTS subjects 
tested (reading, mathematics, and science). They also earned higher scores on the state English Language Arts (ELA) and 
mathematics tests. 

2. Students who attended participating network schools reported more positive interpersonal and intrapersonal outcomes 
than students who attended non-network schools. In particular, they reported higher levels of collaboration skills, academic 
engagement, motivation to learn, and self-efficacy. There were no significant differences between students who attended network 
and non-network schools on reported creative thinking skills, perseverance, locus of control, or self-management. 

3. Students who attended participating network schools were more likely to graduate from high school on time (within four years 
of entering Grade 9) than were students who attended non-network high schools. The graduation rate among students who 
attended network schools was estimated to be about 9 percentage points higher than among similar students who attended 
non-network schools. 

4. Students who attended participating network schools and non-network schools had similar rates of enrollment in postsecondary 
institutions overall. However, students who attended network schools were more likely to enroll in four-year institutions and in 
selective institutions. 

5. Although there were significant positive effects of attending a network school averaging across the pairs of network and 
non-network schools in our sample, for many outcomes-for example, PBTS mathematics scores-the effects of attending 
a network school varied significantly across individual pairs of schools. 

Attending a network school had similar benefits for students who entered high school with low achievement and those who entered 
with high achievement, particularly for the test score and high school graduation outcomes. However, while attending a network school 
increased the postsecondary enrollment rate of students who entered high school with low achievement, it had no effect on the 
postsecondary enrollment rate of students who entered with high achievement. 
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Introduction 


The world high school students face today is very different from the world experienced by previous 
generations. With the rapid evolution of technology, the global expansion of jobs and businesses, 
and an increasingly complex and diverse democracy, new graduates must navigate an environment 
that is rapidly and continually changing. Despite decades of educational reform efforts, concerns 
persist that too few American students acquire the knowledge and skills required to become 
engaged and productive citizens of this changing world (Murnane & Levy, 1996; Levy & Murnane, 
2013). These concerns are more pronounced in schools that serve disproportionate numbers of 
students of color and students living in poverty. In response, a movement in support of “deeper 
learning” has emerged among researchers, policymakers, and practitioners in an effort to improve 
students’ future success in college and in their careers and civic life. 

The concept of deeper learning has been used both to describe a set of competencies or goals for 
students and to characterize a way of learning (or a process) that promotes these competencies. 
The William and Flora Hewlett Foundation — a leader in the national initiative to promote deeper 
learning in schools — has defined deeper learning as “a set of competencies students must 
master in order to develop a keen understanding of academic content and apply their knowledge 
to problems in the classroom and on the job” (William and Flora Hewlett Foundation, 2013, p. 1). 

In this view, deeper learning focuses on the development of six interconnected competencies that 
many argue are prerequisites for success in college, career, and civic life: 

• Mastery of core academic content 

• Critical thinking and complex problem-solving skills 

• Effective communication skills 

• Collaboration skills 

• An understanding of how to learn 

• Academic mindsets (William and Flora Hewlett Foundation, 2013; Chow, 2010; Trilling, 2010). 

Taking a slightly different approach, a recent review of theory and research across an array of 
disciplines led a National Research Council panel (National Research Council [NRC], 2012) to 
define deeper learning as “the process through which an individual becomes capable of taking 
what was learned in one situation and applying it to new situations (i.e., transfer).” The panel 
distinguished that process from the specific “21st century competencies” it produces. The NRC 
grouped these competencies into three domains: the cognitive domain, the interpersonal domain, 
and the intrapersonal domain. These domains neatly subsume the six dimensions identified 
by the Hewlett Foundation, providing a compatible framework for the purposes of both research 
and practice. 
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Evidence of Deeper Learning Outcomes 


Proponents of deeper learning argue that core content knowledge is insufficient without broader 
thinking and application skills. Sutherland, Shin, and Krajcik (2010), for example, state: 

“It is not enough for students only to understand big ideas; in fact, they cannot develop integrated 
understandings of even these core ideas unless they use their knowledge in meaningful ways, 
applying what they know to a variety of contexts and to novel situations” (p. 4). Further, proponents 
argue that exposure to instruction focused on deeper learning prepares students to be successful 
professionals, thinkers, and citizens in their adult lives (Finegold & Notabartolo, 2010). Specifically, 
they argue that supporting students to acquire both academic content knowledge and the skills 
required to critically deploy this knowledge facilitates the development of “competencies that 
enable graduating high school students to be college and career ready and then make maximum 
use of their knowledge in life and work” (Alliance for Excellent Education, 2010, p. 1). 

Unfortunately, the research on deeper learning has lagged behind the political and educational 
interest in this concept and the activity of practitioners in schools and districts. In particular, there 
is an absence of empirical research that evaluates whether instruction focused on deeper learning 
affects students’ educational experiences and outcomes (NRC, 2012). 1 Although early evaluation 
studies of schools participating in networks focused on deeper learning suggested positive effects, 
the studies had a number of limitations relating to their research designs, samples, data, measures, 
and/or analyses (Yuan & Le, 2010). More recent evaluations (Collins et al., 2013; Guha et al . , 
2014; Nichols-Barrer & Haimson, 2013) have also suggested positive program effects on 
indicators such as grade point average (GPA), progress to graduation, and state test results, 
but these studies are primarily descriptive in nature or have focused on demonstrating the 
effectiveness of specific instructional programs or approaches aligned with the goals of deeper 
learning. Indeed, a recent NRC panel noted the limitations of existing (primarily correlational) 
research in establishing linkages between deeper learning competencies and long-term outcomes 
for students and recommended that foundations and federal agencies support further research in 
this arena (NRC, 2012). As a result of this limited empirical base, there has recently been increased 
interest in rigorous research that evaluates whether school approaches explicitly focused on 
developing deeper learning competencies are associated with improved educational experiences 
and outcomes for students from all backgrounds. 

The Study of Deeper Learning: Opportunities and Outcomes — funded by the Hewlett Foundation — 
aimed to determine whether students who attended high schools with a mature and at least 
moderately well implemented approach to promoting deeper learning actually experienced greater 


1 Emerging research suggests that exposure to instruction focused on deeper learning correlates with increased academic 
achievement, leading to a more flexible and competent relationship with knowledge (Darling-Hammond, 2011; Alexander, 
Dinsmore, Grossnickle, List, Loughlin, & Parkinson, 2010; Finegold & Notabartolo, 2010; Sutherland, Shin, & Krajcik, 2010; 
Wickersham & McGee, 2008). Most recently, researchers at the Educational Policy Improvement Center (Collins et al., 2013) 
examined the impacts of a curriculum intended to promote “deeper learning” — the Roadtrip Nation Experience (RTN) — and 
found increases in the GPA of RTN students compared to their peers. However, all of these studies note limitations or lack 
methodological documentation. 
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deeper learning opportunities and outcomes than likely would have been the case had they not 
attended these schools. 2 In contrast to an evaluation of a particular program or approach to 
deeper learning, this proof-of-concept study focused on providing evidence to indicate whether 
schools can promote deeper learning across a variety of reasonably well implemented approaches 
and a diversity of students. This study aimed to address the evidence gap related to deeper learning 
by using a rigorous quasi-experimental design 3 to examine a set of high schools (hereafter referred 
to as “network” schools) associated with ten established networks from across the country that 
embrace the goals of deeper learning, promote instructional practices they believe are likely to 
lead to deeper learning competencies, and participate in the Hewlett Foundation’s Deeper Learning 
Community of Practice. (See Box 1 for a list of participating networks.) As described in the first 
report in this series, The Shape of Deeper Learning: Strategies, Structures, and Cultures in Deeper 
Learning Network High Schools (Huberman, Bitter, Anthony, & O’Day, 2014), the network schools 
included in this study shared an explicit, school-wide focus on deeper learning as a goal for 
students. While employing a diverse range of approaches to promote deeper learning, they 
had several strategies and structures in common. 

Box 1: Networks Participating in the Hewlett Foundation’s Deeper Learning Community of Practice 

Asia Society - http://asiasociety.org/international-studies-schools-network 
Big Picture Learning - http://www.bigpicture.org/ 

ConnectEd - http://www.connectedcalifornia.org/ 

EdVisions Schools - http://www.edvisions.com/ 

Envision Schools - http://www.envisionschools.org/ 

Expeditionary Learning - http://elschools.org/ 

High Tech High - http://www.hightechhigh.org/ 

Internationals Network for Public Schools - http://internationalsnps.org/ 

New Tech Network - http://www.newtechnetwork.org/ 

New Visions for Public Schools - http://www.newvisions.org/ 

Note: See our first report (Huberman et at, 2014) for more information on the Deeper Learning Community of Practice. 


2 See our first report (Huberman et at, 2014) for a description of the approaches to promoting deeper learning taken by schools 
in this study. 

3 A quasi-experimental design estimates the effect of a “treatment,” program, or intervention by comparing outcomes for people 
who chose or were selected to participate and those who did not, rather than by randomly assigning participants (see Shadish, 
Cook, & Campbell, 2002). 
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The Hewlett Foundation’s deeper learning initiative reflects a set of core assumptions about how 
a school’s approach to promoting deeper learning can result in improved outcomes for students, 
including postsecondary success and civic and employment outcomes. In this study, we examined 
the underlying premise of this initiative: that explicit strategies, structures, and school culture 
designed to support deeper learning would result in students experiencing greater opportunities 
to engage in deeper learning. These opportunities would, in turn, lead to transferable knowledge 
and skills — or competencies — that are critical to readiness for college, career, and civic life. 

For analysis and interpretation, we group these competencies into three overlapping domains, 
as defined by the National Research Council (2012): the cognitive domain, including mastery of 
academic content knowledge and complex problem solving; the interpersonal domain, including 
collaboration and communication skills; and the intrapersonal domain, including an understanding 
of how to learn and academic mindsets such as motivation to learn, academic engagement, and 
self-efficacy (Farrington et al. , 2012; Soland, Hamilton, & Stecher, 2013; NRC, 2012). Proponents 
of deeper learning argue that approaches focused on developing these competencies can improve 
outcomes for all students, including those from traditionally underserved groups and those who 
have not previously experienced educational success. The abbreviated theory of action for the 
deeper learning initiative (shown in Exhibit 1) delineates the key hypothesized relationships 
between school approaches to promoting deeper learning, opportunities to engage in deeper 
learning, and outcomes. In this graphic, we provide additional detail related to the focus of this 
report — students’ outcomes. 

Our first report from this study described key aspects of participating schools’ approaches to 
promoting deeper learning (Huberman et al., 2014) — that is, what adults in network schools did to 
develop deeper learning competencies. In that report, we described the strategies and structures 
network schools used to foster student development in the three competency domains (cognitive, 
interpersonal, and intrapersonal). The findings in our first report are summarized in Box 2a. 
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Exhibit 1: Abbreviated Theory of Action 4 



4 Not all of the outcomes included in this diagram are measured in this study through survey and assessment data (for example, 
the "understanding how to learn” outcome). Many of the intrapersonal outcomes shown in the diagram align with the sixth 
deeper learning competency — the development of academic mindsets. 
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Box 2a: Key Findings From The Shape of Deeper Learning: Strategies, Structures, and Cultures in Deeper 
Learning Network High Schools 


1. Across the three deeper learning domains, sampled network schools used a range of strategies to 
develop deeper learning competencies-most commonly project-based learning, internship opportunities, 
collaborative group work, and longer-term cumulative assessments. 

• In the cognitive domain, all but one network school (18 of 19) employed project-based learning (PBL) 
to some degree to develop mastery of core academic content knowledge and critical thinking skills. 
PBL was integral to daily instruction in slightly over a third of these schools and used more sporadically 
in others. 

• Also in the cognitive domain, three quarters of the network schools (14 of 19) provided connections to 
the real world through internship opportunities for students. At two schools, internships were considered 
central to learning and occurred two or three days per week across all four years. The remaining 12 schools 
incorporated internships for a portion of students at some point in their school career to provide career- 
related experience, boost life skills, or help with the transition from high school to college and careers. 

• In the interpersonal domain, collaboration and communication skill development was an explicit 
goal reported by staff at a majority of network schools (11 of 19), which they addressed through 
collaborative group work and longer term assessments (such as portfolios and exhibitions, where 
students had to present and defend their work). 

• In the intrapersonal domain, almost half of the network schools (9 of 19) reported having explicit goals 
related to intrapersonal competencies (learning how to learn and academic mindsets) for students and 
they used a variety of strategies to encourage the development of these skills, including study groups 
and student participation in decision making. Three schools focused on individualized learning as a 
way to develop independent learning and self-management skills. 

2. Most network schools supported the implementation of instructional approaches aligned with deeper 
learning through the development of specific structural and cultural elements, including advisory classes 
(16 schools), alternative scheduling (14 schools), and personalized learning environments (all schools). 
However, these structures and cultures looked different across the schools. For example, advisory classes 
had different numbers of students (from 15 to 30 students), ran for different amounts of time (between 
30 and 60 minutes), and happened with different frequencies (from every day to once or twice a week), 
depending on the school. 

3. Comparisons between the network and non-network school principal interview data suggest that the 
network schools employed strategies to foster the deeper learning competencies to a greater extent than 
did the non-network schools, particularly in the areas of project-based learning, internship opportunities, 
collaborative group work, longer term cumulative assessments, and development of intrapersonal skills. 
Network schools also employed advisory classes and alternative scheduling to a higher degree than the 
non-network schools. 


In the second report — Providing Opportunities for Deeper Learning — we focused on the relationship 
in the theory of action between school approaches to promoting deeper learning (including 
strategies, structures, and cultures) and the opportunities that students have to engage in 
deeper learning (Bitter, Taylor, Zeiser, & Rickies, 2014). The results indicated that students 
who attended network schools had more opportunities to engage in deeper learning than they 
would have had they not attended the network schools. (The findings in our second report are 
summarized in Box 2b.) 
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Box 2b: Key Findings From Providing Opportunities for Deeper Learning 


1. On average, students who attended the network schools in the study reported greater opportunities to engage 
in deeper learning than did similar students who attended non-network schools. 

2. Since the study schools served substantial populations of high-poverty students and, in some cases, large 
populations of English learners, the results demonstrate that these opportunities were provided to a diverse 
group of students, including traditionally underserved subgroups of students. 

3. The effects of attending a participating network school on deeper learning opportunities were similarly 
positive for subgroups of students including initially high- and low-achievers and students who did or did 
not qualify for free or reduced price lunch. 

4. Teachers’ most challenging assignments collected from the network schools exhibited greater opportunities 
for independent learning in mathematics and real-world connections in ELA than the challenging 
assignments collected from the non-network schools, but were not significantly different on other 
opportunity measures (including complex problem solving, communication, and conceptual understanding 
in mathematics). 

5. The opportunities for deeper learning experienced by individual students, regardless of whether they attended 
a network school, were associated with those students’ deeper learning outcomes. 


In this third report, we focus on whether students who attended selected network schools had 
higher educational outcomes than did their matched counterparts who attended similar non- 
network schools. In particular, we focus on two key questions: 

1. Did students who attended the selected network schools achieve better outcomes than 
would likely have been the case had they not attended the network schools? In particular, 
did they demonstrate improved outcomes in the following domains (described in the theory 
of action): 

a. Cognitive competencies 

b. Interpersonal and intrapersonal competencies 

c. High school graduation 

d. Postsecondary attendance, persistence, and selectivity 

2. Were the impacts of attending a participating network high school similar for all students 
who attended these schools, or did certain subgroups of students (e.g., students with low 
and high levels of prior achievement) differentially benefit from attending these schools? 

Study Design 

To address the study’s research questions, we examined a set of high schools that belong 
to 10 established networks from across the country that embrace the goals of deeper learning, 
promote instructional practices that they believe are likely to develop deeper learning competencies, 
and participate in the Hewlett Foundation’s Deeper Learning Community of Practice (network 
schools). (See Box 1 for a list of the networks.) 
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Given that we were conducting a “proof of concept” study (as opposed to an evaluation of network 
schools as typically implemented), we included network schools that were considered moderate 
or high implementers of their network's model, based on network representatives’ reports and 
rubrics. All schools were nonselective and served substantially disadvantaged populations. In 
addition, all schools included in the analyses in this report were located in California or New York 
City. 5 (See Box 3a for additional details regarding the selection process for schools and students.) 

To examine the effects of attending each network school on student outcomes, we selected a 
matched comparison school for each network school. These comparison schools (hereafter 
referred to as “non-network” schools) were chosen because they were located in the same 
geographic area as the network school they were matched with, and because they had a similar 
incoming student population (based on student demographics and achievement). We chose each 
non-network school as a proxy for the schools that students attending the network school might 
have attended had they not attended the network school. The non-network schools provided a 
basis for comparison, allowing us to infer whether attending a network school resulted in better 
outcomes for students. The quality of the conclusions about the effects of attending a network 
school hinges on whether the students attending its matched non-network school were similar 
to the students attending the network school on measured and unmeasured background 
characteristics (e.g., prior achievement and motivation) at the time of entering Grade 9. To 
compensate for any difference in measured background characteristics between students in 
network schools and matched non-network schools, we weighted the students in each 
comparison school so that, once weights were applied, the distribution of measured student 
characteristics in the non-network school reflected the distribution of student characteristics in 
the network school. (See the Technical Appendix, section IV.A for an explanation of the weights). 

Exhibit 2 provides a summary of the characteristics of network and non-network schools in 
the sample. 

Exhibit 2: Characteristics of Network and Non-Network Schools Participating in the Study of Deeper Learning 



Source: National Center for Education Statistics, from the 2010-11 Common Core of Data. 


5 Although three of the schools included in our first report (Huberman et al., 2014) were located in other states, the sample for 
the analysis of student opportunities and outcomes discussed in our second report (Bitter et al., 2014) and in this report was 
restricted to California and New York City because we were unable to identify appropriate matched non-network schools for the 
three schools located in other states. 
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Box 3a: Participating Schools and Students 


The sample of schools included high schools participating in one of the 10 networks in the Hewlett Foundation’s 
Deeper Learning Community of Practice and identified as moderate or high implementers of their network's 
approach. In addition, the sample included a set of comparison non-network schools serving similar student 
populations but not participating in any of the 10 networks. While the non-network schools were not members 
of the 10 networks, and deeper learning was not a focus at these schools, they may have been implementing 
other reforms. 

Selecting Schools: We used several criteria to choose high schools from the participating networks (network 
schools). We selected schools that had implemented the network approach schoolwide and were considered 
to be moderate or high implementers of the approach, according to network representatives (based on criteria 
established by each network). In addition, we sought schools that had been in existence for at least four years 
(i.e., long enough to have graduated at least one class by the start of the study); that were non-selective in 
admissions (increasing the validity of comparisons between students in network and non-network schools); 
and that enrolled at least 200 students (ensuring a sufficiently large sample of students for data analyses). 
Because we were particularly interested in outcomes for economically disadvantaged students, we sought 
schools in which at least 25 percent of students were eligible for free or reduced-price lunch, although most 
of the schools we selected had a substantially larger percentage of eligible students (see Technical Appendix, 
section II. A). 

We identified a matched non-network comparison school for each network school using data from the 
Common Core of Data (CCD), as well as aggregated student-level data obtained from each district. Student- 
level district data, which we aggregated to the school level, included student demographic and socioeconomic 
characteristics, as well as incoming student test scores (i.e., Grade 8 mathematics and English Language Arts 
test scores). For each network school, we recruited a non-network school in the same district or in a neighboring 
district. We sought non-network schools that, like the network schools, had been in existence for at least four 
years, were non-selective in admissions, and enrolled at least 200 students, at least 25 percent of whom 
were eligible for free or reduced-price lunch. 

The network schools included in the analyses described in this report represent 8 of the 10 networks and 
were located in five different districts across two states: California and New York. The non-network schools 
were located in six districts across these two states.* 

All analyses described in the report were conducted based on matched pairs of network and non-network 
schools. The number of school pairs included in each analysis varied depending on the availability of data. For 
students’ interpersonal and intrapersonal competencies (measured in the student survey), 11 pairs of schools 
had sufficient numbers of students who consented to participate in the survey. For the OECD PISA-Based Test 
for Schools (PBTS), 10 pairs of schools had sufficient numbers of students who consented to participate. 
For high school achievement and high school graduation, data were available for 13 school pairs. Finally, 
postsecondary data were obtained and analyzed for 11 school pairs. 

Selecting Students: As described in Box 3b, the study focused on four cohorts of students who had been in 
the district since Grade 8 (allowing us to measure prior levels of achievement and demographic characteristics) 
and who entered Grade 9 in one of the sampled schools between the 2007-08 and 2010-11 academic 
years. In addition, for the PBTS and survey outcomes, analyses were restricted to students with parental 
consent to participate in the study. In some cases, challenges associated with obtaining active consent 
(required by four districts) limited the number of students who were permitted to participate in primary data 
collection activities. We used several approaches to maximize the number of students included in the data 
collection activities and analyses, and we accounted for non-response in the analysis model (see Technical 
Appendix, section II. B). 


* We included 16 network schools and 12 non-network schools. Two of the network schools were combined for analyses because they 
had small student populations eligible for the study and were co-located on one campus. We refer to this as one school, and thus the 
sample is based on 15 network schools. To take advantage of the available pool of non-network schools, 3 of the 12 non-network 
schools were matched with two network schools each, resulting in a total of 15 school pairs. 
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To assess the effects of attending a network school, we collected multiple outcome measures for 
four cohorts of incoming Grade 9 students. (See Box 3b.) To measure cognitive competencies, we 
relied on the OECD PISA-Based Test for Schools (PBTS), which measures students' critical thinking 
and problem-solving skills, as well as their mastery of core content in reading, mathematics, and 
science. We also obtained students' scores on state assessments in English Language Arts (ELA) 
and mathematics. To measure interpersonal and intrapersonal outcomes, we relied on survey 
measures of collaboration skills, creative thinking skills, academic engagement, motivation to 
learn, perseverance, self-efficacy, locus of control, and self-management. To measure students’ 
academic success in high school, we relied on data from school districts on whether students 
graduated on time. Finally, to measure postsecondary outcomes, we gathered data from the 
National Student Clearinghouse on student enrollment and persistence in postsecondary 
institutions. (See Box 4 for a description of the study’s data sources.) 

Box 3b: Student Cohorts 


The analyses of deeper learning outcomes presented in this report are based on four student cohorts: 

• Cohort 1: Students who entered Grade 9 in 2007-08 (expected graduation in spring 2011; in second year 
of college in spring 2013 if progressed at normative pace) 

• Cohort 2: Students who entered Grade 9 in 2008-09 (expected graduation in spring 2012; in first year 
of college in spring 2013 if progressed at normative pace) 

• Cohort 3: Students who entered Grade 9 in 2009-10 (expected to be in Grade 12 in spring 2013) 

• Cohort 4: Students who entered Grade 9 in 2010-11 (expected to be in Grade 11 in spring 2013) 
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Box 4: Data Sources 


The analyses of deeper learning outcomes presented in this report rely on four primary data sources: 

1. A Student Survey: We administered a one-hour survey to participating students in spring 2013. The survey 
was designed to measure: 1) the opportunities students experienced in school related to the deeper 
learning competencies, and 2) the interpersonal and intrapersonal outcomes that are hypothesized to 
be important for college and career readiness in our theory of action. We administered the survey to all 
sampled and consented students in Grade 11 (Cohort 4) and Grade 12 (Cohort 3) in each participating 
network and non-network school. In total, we obtained completed surveys for 1,762 students in 11 pairs 
of schools, with an average response rate of 76 percent among sampled students (ranging from 54 percent 
to 93 percent of sampled students, by school).* 

The survey included previously validated item sets from national surveys, including the Consortium of 
Chicago School Research (CCSR) survey and the High School Longitudinal Survey (see the Technical 
Appendix, section III.C for a full list of items and sources). We supplemented these existing items with 
original items designed to address specific constructs important to this study. The survey was piloted prior 
to administration to test the validity of the scales. We calculated scale scores for the survey constructs 
using a Rasch analysis.** These Rasch scores were standardized using the mean and standard deviation 
among non-network students so that the results could be presented as effect sizes. 

2. The OECD PISA-Based Test for Schools (PBTS): For the purposes of this study, CTB McGraw Hill 
administered the PBTS to the students who were selected to complete the student survey. The PBTS 
was administered during two consecutive one-hour testing sessions in spring 2013. We administered the 
test to all sampled and consented students in Grade 1 1 (Cohort 4) and Grade 12 (Cohort 3) in each 
network and non-network school. In total, we obtained PBTS scores for 1,267 students in 10 pairs of 
schools, with an average response rate of 61 percent among sampled students. 

3. Extant, Student-Level Data: We obtained extant, student-level demographic data from the districts 
associated with the participating schools for five cohorts of students. We used these data to calculate 
propensity scores for students and to select student samples. Demographic data and students’ Grade 8 
achievement test scores were also used as covariates in the analysis models. In addition, school districts 
provided data regarding high school graduation and students’ performance on state achievement tests by 
fall 2013. High school graduation data was obtained for students in Cohorts 1-3 (for a total of 13,831 
students within 13 pairs of schools) and high school achievement test score data was obtained for 
students in Cohorts 1-4 (for a total of 13,960 students within 12 pairs of schools for mathematics and 
14,306 students within 13 pairs of schools for ELA). 

4. Postsecondary Outcomes From the National Student Clearinghouse: We requested postsecondary 
outcome data through the National Student Clearinghouse-a non-profit organization that collects 
postsecondary enrollment and completion data from over 3,600 participating colleges— that contain 
information about approximately 98 percent of all students enrolled in postsecondary education.*** 
Postsecondary outcome data were obtained for students in Cohorts 1-3, who entered Grade 9 between 
the 2007-08 and 2009-10 academic years, for a total of 11,165 students within 11 pairs of schools. 


* A technological complication with online survey administration resulted in a large amount of missing data in one of the non- 
network schools. For survey constructs affected by this technological issue, analyses excluded this pair of schools. (See Technical 
Appendix, section III.C for more details.) 

** Rasch analysis is a method of generating scale scores on a survey or test based on responses to individual items. 

*** http://www.studentclearinghouse.org/about/clearinghouse_facts.php 
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To estimate the effects of attending a network school on student outcomes, we conducted separate 
analyses for each network school and its paired non-network school. For each pair, we estimated 
the difference in outcomes between students who attended the network school and those who 
attended the non-network school, adjusting for differences in the background characteristics 
of students attending each school. We then averaged the effects across the pairs to obtain an 
overall estimate of the differences between students who attended the network and non-network 
schools in the study. (See Box 5 for more detail on the analysis.) The overall estimates describe 
effects for the particular group of network schools included in the study. Because the schools 
were purposefully selected to be moderately or high implementing (according to their networks) 
and to meet other criteria, the results cannot be generalized to all schools within the 
participating networks. 

Box 5: Analysis Models 

To estimate the differences in deeper learning outcomes for students in network and non-network schools, 
we used doubly robust regression models.* We balanced student characteristics within pairs of schools using 
propensity score weighting (described in the Technical Appendix, section IV.A), and we used weights to account 
for attrition, non-consent, and non-response. We performed student-level analyses separately within each pair 
of network and non-network schools, combining data from the cohorts for which data were available, and 
taking potential differences across cohorts into account in the analysis, as well as other student characteristics 
measured prior to entry into high school. 

We used a meta-analytic technique to estimate the average difference between network and non-network 
students across pairs. As discussed above (Box 4), the student cohorts and pairs included in specific analyses 
differ based on data availability. (See the Technical Appendix, section IV for a more detailed description of the 
analysis methods.) 

We used parallel models for each of the main outcome measures: PBTS scores, achievement scores on state 
tests, interpersonal and intrapersonal outcomes measured using the student survey, high school graduation, 
and postsecondary outcomes. When estimating differences in PBTS scores for students in network and 
non-network schools, we used a regression analysis method that takes measurement error in the PBTS into 
account. For high school graduation and postsecondary outcomes, we used doubly robust logit regression. 


* The analysis is considered doubly robust (Funk et al., 2011) because it accounts for preexisting differences between network and 
non-network students in two ways: first, by adjusting for how student differences are associated with network school selection (using 
propensity score weighting); and second, by adjusting for how student differences are associated with outcomes (using regression 
models). If either of the two adjustment methods accurately accounts for student differences, we can obtain valid estimates of the 
network school’s effect. 

Key Findings 

In this section, we summarize our findings for the four key domains: cognitive competencies; 
interpersonal and intrapersonal competencies; high school graduation; and postsecondary 
attendance. 
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Cognitive Competency Outcomes 


Did Students Who Attended the Network High Schools Score Higher on the 
OECD PISA-Based Test for Schools (PBTS) Than Students Who Attended the 
Non-Network Schools? 

Mastery of core content in academic classes and the application of critical thinking and 
problem-solving skills are two of the competencies that define deeper learning. While high 
school achievement tests measure students' knowledge of subject area content, researchers 
and practitioners have debated whether these exams test students’ ability to think critically. To 
measure students' mastery of both core content and critical thinking skills, we administered the 
PBTS to eligible students in spring 2013, when the sampled students were in Grades 11 and 12. 
(See Box 6.) We administered the assessment to these grade levels — rather than the traditional 
age sample for the PBTS, which typically includes students in Grades 9 and 10 — because they 
had experienced longer exposure to the networks’ deeper learning approaches. 6 The PBTS is 
based on the internationally recognized PISA and provides measures of students' content 
knowledge and application of higher-order thinking skills in reading, mathematics, and science. 

Box 6: Cognitive Competency Outcome Measures 

The OECD PISA-Based Test for Schools (PBTS): We administered the PBTS to examine two key deeper 
learning outcome dimensions: critical thinking and mastery of content knowledge. For the purposes of this 
study, we contracted with the approved U.S. vendor to administer the PBTS to all sampled and consented 
students in Grades 11 and 12 in network and non-network schools in spring 2013. In total, we administered 
the assessment to 570 students in network schools and 697 students in non-network schools, with an 
average response rate of 61 percent. 

California and New York State Achievement Tests: We obtained state test scores for students in Cohorts 1-4. 

In California, we examined students' scores on the California High School Exit Examination (CAHSEE), which 
students take in Grade 10. In New York City, we examined students' test scores on the Comprehensive English 
and Integrated Algebra Regents tests. The New York City tests were administered to students completing specified 
courses, and the grade level at which students took the test depended on the grade in which they took the 
course. Most students in the New York City sample took the Integrated Algebra test in Grade 9 and the ELA test 
in Grade 11. If a student took the test more than once, we used the result from the first administration. 

In both California and New York, students were not required to demonstrate mastery of content beyond algebra 
at the time the tests were administered, and thus the tests do not address content beyond algebra. Test scores 
were standardized based on the California statewide and New York City-wide means and standard deviations 
associated with the year students took the exam so that the results could be analyzed across academic years 
and across states. 


6 We also administered the assessment to the traditional age sample for the PBTS (15-year-old students) in a subset of schools, 
but we did not use these data in the analysis because the students had been at the schools for only a short amount of time. The 
results for the traditional sample were provided to the schools so that they could examine their students’ performance against 
established international benchmarks and performance patterns. 
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Students who attended participating network high schools achieved higher scores on the PBTS 
than did comparison students with similar baseline background characteristics, including prior 
state test scores. Students attending network schools scored higher in all three subjects tested 
(reading, mathematics, and science) with effect sizes of 0.12, 0.11, and 0.11 of a standard 
deviation, respectively. 7 (See Exhibit 3.) One way to assess the magnitude of these effect 
estimates is to compare them with the average difference in PBTS scores for students in 
Grades 11 and 12 — the two grades included in our analyses. In reading, Grade 12 students 
in our sample scored about 0.11 standard deviations higher than students in Grade 11. 8 In 
mathematics, Grade 12 students scored 0.16 standard deviations higher than Grade 11 students, 
and they scored 0.12 standard deviations higher than Grade 11 students in science. Thus, the 
effect of attending a network school is about the same as the average improvement in PBTS 
scores from Grade 11 to Grade 12 in reading and science, and about two thirds of the average 
improvement between grades in mathematics. 


Exhibit 3: Estimated Average Effect of Attending a Network School on Students’ Cognitive Competency Outcomes 
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Note: The plotted points represent the meta-analytic average effect estimate for each PBTS and high school achievement score 
(see Technical Appendix, section IV.B for a detailed description of the analytic methods), and the vertical bars represent each estimate’s 
95 percent confidence interval. Effect sizes are significant when the full confidence interval lies above or below the zero line (all measures 
in this chart). 


7 One way to interpret an effect size is to translate it into percentile points. For example, an effect size of .12 implies that a network 
student at the 50th percentile on the PBTS would have scored at the 45th percentile if he or she attended a non-network school. 

8 These estimates control for background characteristics that may differ for students in Grades 11 and 12 in our sample, including 
prior achievement, gender, and treatment status. 
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Another way to put these effects into context is to compare them with the results that have been 
obtained for other high school reform efforts. The results are similar in magnitude to the effects 
obtained for other reforms that have been rigorously evaluated and have shown positive effects. 

For example, a quasi-experimental study of Talent Development High Schools found an impact 
of 0.12 standard deviations on mathematics achievement (Kemple, Herlihy, & Smith, 2005) and 
a randomized study of Early College High Schools found an impact of 0.14 standard deviations on 
high school ELA achievement (Berger et al., 2013). A study of the impact of attending Expeditionary 
Learning middle schools found an effect of 0.11 standard deviations on achievement in reading 
and 0.09 standard deviations in mathematics among students in Grades 7 and 8 (Nichols-Barrer 
& Haimson, 2013). 

Did Students Attending the Network High Schools Score Higher on State 
Achievement Tests Than Students in the Non-Network Schools? 

Students who attended participating network high schools achieved higher scores on the state 
ELA and mathematics tests than did comparison non-network students with similar baseline 
characteristics. (See Exhibit 3). While some have argued that a focus on deeper learning may 
not improve students’ performance on standardized state tests because the tests do not assess 
depth of knowledge or other deeper learning skills, these tests continue to be an important facet 
of education policy. As a result, we examined whether students who attended participating network 
high schools had similar or higher high school achievement test scores in ELA and mathematics 
compared to similar students attending non-network schools. We found that the impact of attending 
a deeper learning network school on achievement test scores in mathematics was similar to the 
effect sizes observed for PBTS scores (0.10 standard deviations), while the impact on ELA test 
scores was smaller in magnitude (0.05 standard deviations). 

Did the Effects on Students’ Cognitive Outcomes Differ Across Network Schools? 

Looking beyond the average effect, we found significant variation across pairs of network and 
non-network schools in the effects of attending a network school, especially for mathematics 
and science. The estimated effect of attending a network school on PBTS reading, mathematics, 
and science scores was positive and significant in some pairs, and not significant in others. 
There were no significant negative effects. The result for each outcome reported above is based 
on the average across the pairs of network and non-network schools included in the study. As 
described in our first report (Huberman et al., 2014), the network schools took a variety of 
approaches to fostering deeper learning, and they were located in a variety of district contexts. 
As such, it might be anticipated that they would vary in their effects. 
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The study was not designed to obtain precise estimates of effects for each network school, given 
that the number of Grade 11 and Grade 12 students enrolled in each school was relatively small, 
meaning that some of the apparent variation in effects across pairs is due simply to chance. 
However, an examination of the individual pair results provides some indication of the degree 
to which the average results presented above are typical across the pairs of network and 
non-network schools in the study. Exhibit 4, for example, displays the estimates for the 10 pairs 
included in the analysis of PBTS mathematics scores. The estimates were positive and significant 
for 3 of the 10 pairs, positive but not significant for three pairs, and negative but not significant for 
four pairs. Given the available precision, we cannot be certain about the exact magnitude of the 
effect for each individual school, but we can be sure that the true impacts differ from school to 
school. The estimates were similar for science and were less varied for reading. (See the Technical 
Appendix, section V.A. 9 ) 


Exhibit 4: Estimated Average Effect of Attending a Network School on Students’ PBTS Mathematics Achievement, 
by Individual School Pair 
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Note: The plotted points represent the effect estimate for PBTS mathematics for each school pair included in the meta-analysis. The 
horizontal bars represent each estimate’s 95 percent confidence interval. Effect sizes are significant when the full confidence interval lies 
entirely to the right or to the left of the zero line. 


9 See the Technical Appendix, section V.B, for the individual pair results for the state assessment. 
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Did the Effects on Students’ Cognitive Outcomes Differ for Student Subgroups? 


One of the network schools’ goals is to foster deeper learning among all students, not just those 
traditionally identified as high achieving or college-bound. To examine whether this goal was met, 
we examined whether results differed among students who entered Grade 9 with lower ELA 
achievement and students who entered Grade 9 with higher ELA achievement. 10 

The effects of attending a network school on PBTS scores did not differ among students 
who entered high school with lower and higher prior achievement. For schools in California, 
we conducted the analysis by comparing students who entered high school with ELA test scores 
below the average for their state with students who entered high school with test scores above 
average for their state. For schools in New York City, we compared students below and above 
average for their district. 11 The results suggest that both students who entered high school with 
below-average scores and students who entered with above-average scores benefited from attending 
a network school. The results were similar for state tests. 12 (See the Technical Appendix, section V.F.) 

Interpersonal and Intrapersonal Competency Outcomes 

Scores on the PBTS and high school achievement test scores address two of the six deeper 
learning competencies: mastery of core content and the application of critical thinking and 
complex problem-solving skills. We measured other competencies associated with deeper 
learning through the student survey. As described in Box 7, items in the student survey 
measured students’ creative thinking skills, collaboration skills, academic mindset (i.e., 
academic engagement, motivation to learn, perseverance, locus of control, and self-efficacy), 
and understanding of how to learn (i.e., self-management). The survey measures were based on 
students' self-reports of their skills — an approach commonly used to tap these kinds of skills. 


10 We conducted this analysis using students’ ELA scores because Grade 8 mathematics scores in California were based on different 
subject-specific mathematics tests, and thus it was difficult to determine whether a student was below or above average. 

11 For both California and New York, we also conducted a parallel analysis comparing students who entered high school with scores 
below or above average for their pair, and the results were similar. 

12 We also conducted similar analyses for students from economically disadvantaged backgrounds (as indicated by eligibility for free 
and reduced-price lunch) and students from more advantaged backgrounds; and for males and females. In addition, for the PBTS, 
we compared results for Grade 11 and Grade 12 students to see whether the benefits of attending a network school improved as 
students progressed through high school. The results of these analyses show no differences in the effects of attending a network 
school across these subgroups for the PBTS and state achievement test scores. (See the Technical Appendix, section V.F.) 
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Box 7: Interpersonal and Intrapersonal Competencies-Student Survey Measures 


The student survey included eight measures of interpersonal and intrapersonal competency outcomes: 

1. Creative Thinking Skills: The extent to which a student perceives that he or she can think of original 
ideas and solutions. 

2. Collaboration Skills: The extent to which a student perceives that he or she works well in a group 
(e.g., positive personal interactions; the ability to pay attention, share ideas, be prepared, and do their 
part) and cooperates to identify or create solutions. 

3. Academic Engagement: The degree to which a student agrees that he or she has “interest and engagement 
in learning” and participates actively in classroom learning activities. 

4. Motivation to Learn: The degree to which a student is motivated to do well academically and to become 
more knowledgeable, measured by a student’s “perceived importance of coursework as well as preference 
for challenge and mastery goals.” 

5. Self-Efficacy: The degree to which a student tends to view him or herself “as capable of meeting task 
demands in a broad array of contexts.” 

6. Locus of Control: The extent to which a student feels he or she has control over what happens to them 
and their beliefs in their own personal control, powerful others, and chance or fate. 

7. Perseverance: The degree to which a student agrees that he or she maintains effort and interest despite 
failure, adversity, and plateaus in progress. 

8. Self-Management: The extent to which a student feels he or she is able to independently manage their 
work and schedules to meet goals. 

Students were asked to respond to a set of items that asked about the extent to which they agreed with 

different statements. (See the Technical Appendix, section III.C, for a detailed list of items and their sources.) 

Response options ranged from 0 (strongly disagree or never or almost never true) to 3 (strongly agree or 

always or almost always true). 

Analyses for self-efficacy and locus of control included 11 school pairs, while analyses of the remaining 

interpersonal and intrapersonal competencies included 10 school pairs. 


Did Students in the Network High Schools Report Higher Levels of Interpersonal and 
Intrapersonal Competencies Than Students in the Non-Network Schools? 

Students who attended participating network high schools reported greater collaboration skills 
and higher levels of academic engagement, motivation to learn, and self-efficacy than similar 
students who attended non-network high schools. There were no significant differences between 
network and non-network students on reported creative thinking, perseverance, locus of control, 
or self-management. The effect sizes for the statistically significant outcomes ranged from 
0.12 standard deviations (for motivation to learn) to 0.20 standard deviations (for academic 
engagement). (See Exhibit 5.) These effects are somewhat larger than the effects on cognitive 
outcomes, but the evidence base on interpersonal and intrapersonal outcomes is currently too 
limited to judge the magnitude of the results in terms of their educational significance (few other 
studies have examined similar outcomes). A recent evaluation of Talent Development High Schools 
found an impact on student attendance of 0.19 standard deviations (Kemple et al ., 2005), and 
while attendance is not directly comparable to our measures, the study-reported effect size is 
similar to the effects on interpersonal and intrapersonal outcomes observed here. 
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Exhibit 5: Estimated Average Effect of Attending a Network School on Students’ Interpersonal and Intrapersonal 
Competency Outcomes 
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Note: The plotted points represent the meta-analytic average effect estimate for each interpersonal and intrapersonal competency outcome 
(see Technical Appendix, section IV.B for a detailed description of the analytic methods), and the vertical bars represent each estimate's 
95 percent confidence interval. Effect sizes are significant when the full confidence interval lies above or below the zero line. 


Did the Effects on Students’ Interpersonal and Intrapersonal Outcomes Differ Across 
Network Schools? 

The estimated effects of attending a network school on students’ interpersonal and intrapersonal 
competency outcomes varied significantly across pairs of network and non-network schools. 
The estimates were positive and significant for some of the pairs of network and non-network 
schools, and generally not significant for others. There were few significant negative estimates. 

For example, we found that attending a deeper learning network school had a significant positive 
estimated effect on academic engagement in 2 of the 10 pairs of schools included in the analysis; 
positive and not significant effects in six school pairs; and negative but not significant effects 
in two school pairs. For creative thinking — a domain in which there was no significant overall 
effect — 7 of the 10 pair effects were positive, although none were significant (unlike the results for 
academic engagement). In addition, the effect on creative thinking was negative and statistically 
significant in one school pair. For that same pair, we also observed negative effects for perseverance, 
locus of control, motivation to learn, self-management, and self-efficacy. This was the only pair in 
which we observed consistent negative effects on interpersonal and intrapersonal outcomes. 13 
(See the Technical Appendix, section V.C, for more details about results for individual school pairs). 


13 Because the results for this one pair of schools were quite different from the other pairs, they have some influence on the overall 
average reported in Exhibit 5. We retained the pair in the analysis because our goal was to estimate the average effect for the 
pairs in the sample. As a sensitivity analysis, we also estimated the average effects after removing this pair. While these alternative 
results showed larger effects of attending a network school for collaboration skills, academic engagement, motivation to learn, 
and self-efficacy, we did not observe significant effects on creative thinking skills, locus of control, perseverance, or self-management. 
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Did the Effects on Students’ Interpersonal and Intrapersonal Outcomes Differ 
for Student Subgroups? 


We found some indication that attending a network school had more positive effects on 
interpersonal and intrapersonal outcomes for students who entered high school with higher 
achievement than for those who entered high school with lower achievement. In particular, 
attending a network school had more positive effects for high-achieving students on four outcome 
measures: academic engagement, motivation to learn, perseverance, and self-management. (See 
the Technical Appendix, section V.F, for more details about these subgroup analyses.) These four 
outcomes all seem to involve elements of what might sometimes be referred to as “grit.” There 
was no difference between higher and lower achieving students in the effects of attending a 
network school on four other outcomes: creative thinking skills, collaboration skills, self-efficacy, 
and locus of control. 14 

High School Graduation Outcomes 

Were Students Who Attended Participating Network High Schools More Likely to 
Graduate On Time Than Students in Non-Network Schools? 

Even after decades of focus on raising graduation rates (and despite some recent progress), too 
many of America’s youth — particularly those who are disadvantaged — never have the opportunity 
to walk across the graduation stage (Stetser & Stillwell, 2014). With most states now implementing 
college- and career-ready standards that call for more challenging work in high schools, some 
observers fear that these new demands will push more students out of school (Karp, 2014). On 
the other hand, the deeper learning initiative’s emphasis on more ambitious academics, coupled 
with its focus on developing a set of skills that many believe are critical for success in college and 
career, may mean that instruction focused on deeper learning can motivate students to engage in 
their studies, be successful in school, and graduate at higher rates. 

We drew on data provided by the districts in which the network schools were located to determine 
the percentage of students who entered the schools in Grade 9 and graduated on time. (See 
Box 8 for information on the definition of on-time graduation.) 

Students who attended participating network high schools were more likely to graduate 
from high school on time than were comparison students in non-network schools. As shown 
in Exhibit 6, approximately 65 percent of students who attended participating network high schools 


14 Additional analyses examined whether the effects of attending a network school differed among students who were eligible for 
free or reduced-price lunch (FRPL) and those who were not; among males and females; and among Grade 11 and Grade 12 
students. With few exceptions, the results indicate that the effects of attending a network school generally did not differ among 
students eligible for free lunch and those who were not eligible, or among students in Grade 11 and Grade 12. In contrast, the 
findings suggest that the effects on six interpersonal and intrapersonal competency outcomes (collaboration skills, perseverance 
academic engagement, motivation to learn, self-efficacy, and locus of control) were significant and positive for female students 
but were not significant for male students. (See the Technical Appendix, section V.F.) 
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graduated within four years from a high school in the same district. For similar students who 
attended non-network high schools, 56 percent were estimated to graduate within four years 
from a school in the same district. 15 As such, attending a network school increased the on-time 
graduation rate by approximately 9 percentage points. 16 The magnitude of this effect is similar 
to the magnitude observed in a recent evaluation of the impact of attending small high schools in 
New York City. According to the evaluation, among students who entered a lottery to attend a small 
high school, 70.4 percent of lottery winners graduated from high school within four years, compared 
to 60.9 percent of lottery applicants who did not win the lottery to attend a small school (Bloom 
& Unterman, 2014). The effect is also similar to the effect of 8 percentage points obtained in an 
evaluation of Talent Development High Schools (Kemple, Herlihy, & Smith, 2005). 

Box 8: Measure of On-Time Graduation 

We defined students as “on-time graduates’’ if they had a graduation record in the district data system 
within four years of entering Grade 9, including the summer after Grade 12. Any students who did not have a 
graduation record (including students who dropped out, students who took longer than four years to graduate, 
and students who transferred outside of the district or to a private school) were classified as “not on-time 
graduates." We counted students who transferred outside of the participating districts as “not on-time 
graduates” because some of the district data systems did not reliably distinguish students who transferred 
from those who dropped out. 

We were able to identify high school graduates only among students who remained within and graduated from 
participating districts. We conducted two sensitivity analyses to explore how this restriction may have affected 
our estimates of the impact of attending a deeper learning network school on on-time graduation. First, within 
districts where students who transferred to another district could be accurately identified, sensitivity analyses 
examined how estimated effects changed after removing transfer students from the sample, treating the 
transfers as attrition, and applying attrition weights to the students who remained in the district. Second, we 
conducted an analysis that included only students who were still enrolled in the same district in the fall of 
their fourth year of high school, treating students who left prior to the fall of their fourth year as attrition, and 
applying attrition weights to students who were still enrolled at the fall of their fourth year. In both analyses, we 
observed smaller but similarly significant and positive results. We interpret these reported results to be robust 
across our various analytic specifications. 

Graduation rates were computed for students in Cohorts 1-3. Students in Cohorts 4 and 5 would still have 
been enrolled in high school in spring 2013 had they progressed on time. 

The graduation rates reported in the study may be lower than the graduation rates reported by districts and 
states. In particular, the analyses reported in the study are estimates of the graduation rate for students who 
entered the study schools in Grade 9, and students who left the district were counted as not graduating. 
Graduation rates reported by districts and states generally incorporate students who enter school after Grade 
9 and remove students who were identified as transferring out of the school prior to graduation. (See the 
Technical Appendix, sections IV and V for additional information on methods and detailed tables of results.) 


15 As discussed in Box 8, these graduation rates may appear lower than typically reported graduation rates because students who 
transferred to another district prior to graduation were classified as non-graduates. According to the state of California and New 
York City data systems, the published graduation rates for the schools in this study ranged from 48 percent to 100 percent, with 
an average graduation rate of approximately 77 percent. 

16 We also conducted a sensitivity analysis to examine the effects of attending a network school on graduation within five years, 
and the results were similar in magnitude. 
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Exhibit 6: Percentage of Students Graduating From High School Within Four Years of Entering Grade 9 Among 
Students Who Attended Participating Network High Schools and Similar Students Who Attended Non-Network 
High Schools 

Percent of students who graduated 



Note: The percentage for network students is the unadjusted percentage while the percentage for non-network students 
is the adjusted percentage. 

* Difference between network and non-network students is significant at the 0.05 confidence level. 

Did the Effects on Students’ On-time Graduation Differ Across Network Schools 
or for Student Subgroups? 

The estimated effects of attending a network school on graduation varied across pairs of network 
and non-network schools. The estimated effects were positive and significant for 5 of the 13 
pairs of network and non-network schools included in the sample and not significant for the 
others. The estimates were positive but not significant for four school pairs and negative but not 
significant for four school pairs. (See the Technical Appendix, section V.D.) 

The effects of attending a network school on high school graduation did not differ among 
students who entered school with higher or lower achievement. Attending a network school 
improved the odds of graduating to a similar degree for both high and low achievers . 17 (See the 
Technical Appendix, section V.F.) 


17 Additional analyses examined whether the effects of attending a network school differed among students who were eligible for 
free or reduced-price lunch (FRPL) and those who were not; and among males and females. We found that the effects on on-time 
graduation were similar for male and female students, as well as for FRPL-eligible students and students who were not eligible 
for free or reduced-price lunch. (See the Technical Appendix, section V.F.) 
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Postsecondary Attainment Outcomes 


Were Students Who Attended Participating Network High Schools More Likely 
to Enroll or Persist in College Than Students in the Non-Network High Schools? 

A college education has long been associated with greater economic opportunity and financial 
advantage (Card, 1999; Day & Newberger, 2002). In addition, there is evidence that students who 
are less likely to attend college (e.g., low-achieving students, students from low-income families) 
experience the largest financial benefits from their college education (Brand and Xie, 2010). 

The theory of action for the deeper learning initiative hypothesizes that improved cognitive, 
interpersonal, and intrapersonal outcomes, along with increased rates of on-time graduation 
from high school, will lead to increased postsecondary enrollment and persistence. To examine 
these outcomes, we drew on data from the National Student Clearinghouse (NSC). We drew on 
these data to determine the rate at which students who entered Grade 9 in sample schools 
subsequently enrolled in postsecondary education and persisted in college to the fall of the 
second academic year following graduation from high school. (See Box 9 for more information 
on the definitions of postsecondary enrollment outcomes.) 

Box 9: Postsecondary Outcomes 

Using postsecondary data obtained from the National Student Clearinghouse, we defined five 
postsecondary outcomes: 

• Enrolled in any postsecondary institution by fall 2013 

• Enrolled in a two-year institution by fall 2013 

• Enrolled in a four-year institution by fall 2013 

• Enrolled in a selective institution: Selective institutions are identified in the Integrated Postsecondary 
Education Data System (IPEDS) as four-year institutions in which at least 80 percent of students are 
full-time students and the test scores of first-year students place the institution in the top 20 percent 
of institutions in the United States. 

• Persisted into a second year: Students were identified as persisting if they enrolled in postsecondary 
education in the year following expected high school graduation and continued enrollment in the fall of the 
subsequent year. 

Postsecondary enrollment outcomes were measured for students in Cohorts 1-3. Postsecondary persistence 
was measured only for students in Cohorts 1 and 2 (who entered Grade 9 in the 2007-08 and 2008-09 
academic years) because only students in these cohorts had sufficient time to be enrolled for a second year 
of college as of fall 2013. 


Attending a network school did not affect the probability that students enrolled in a 
postsecondary institution, enrolled in a two-year institution, or persisted to the second year 
of postsecondary education by fall 2013. (See Exhibit 7). For example, 51.6 percent of students 
who attended a network school had enrolled in a postsecondary institution by fall 2013, compared 
with 50.4 percent of students who attended non-network schools; and 33.9 percent of students 
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who attended a network school persisted into a second year, compared with 34.8 percent of 
students who attended a non-network school. 

Although attending a network school did not affect overall college attendance rates, students 
who attended participating network high schools were more likely to enroll in four-year 
institutions than were students who attended non-network high schools. Students who 
attended network schools were also more likely to enroll in selective institutions. These 
results may indicate that there is less “undermatching” in network schools than non-network 
schools. Undermatching, as defined by Hoxby and Turner (2013), occurs when high-achieving 
students from at-risk backgrounds do not apply to selective institutions due to a lack of academic 
guidance or misperception of their academic abilities. The higher rate of enrollment in selective 
institutions for students who attended network schools may reflect a reduction in “undermatching" 
brought about by encouraging students to enroll at better colleges that match their capabilities. 
Alternatively, it may be that the higher matriculation rates reflect an increase in “overmatching” 
(enrolling in colleges that are too challenging), which would result in lower rates of persistence 
as students find themselves enrolled in colleges beyond their level of readiness. In order to more 
fully understand this finding, additional research could track cohorts of students into the future 
and examine the types of college courses (e.g., remedial) and majors they complete. Long-term 
tracking of these students (ideally for six years after high school) would allow us to answer 
important questions about whether attending network schools translates into attainment of 
associate’s and bachelor’s degrees and early employment outcomes. 

Exhibit 7: Postsecondary Enrollment and Persistence Outcomes for Students Who Attended Participating 
Network High Schools and Similar Students Who Attended Non-Network High Schools 

Percent of students 



Enrolled Postsecondary Enrolled in two-year Enrolled in four-year Enrolled in selective 

by fall 2013 persistence by fall 2013 by fall 2013 by fall 2013 


Note: The percentage for network students is the unadjusted percentage while the percentage for non-network students is the adjusted 
percentage. Postsecondary persistence is defined as enrollment in the fall of the second year of college. Selective institutions were 
identified in the National Center for Education Statistics (NCES) Integrated Postsecondary Education Data System (IPEDS), based on 
the test scores of incoming freshmen students as well as the percentage of incoming freshmen students who were enrolled full-time. 

* Difference between network and non-network students is significant at the 0.05 confidence level. 
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Did the Effects on Students’ Postsecondary Outcomes Differ Across Network Schools 
or for Student Subgroups? 


The estimated effects of attending a network school varied across pairs for all postsecondary 
outcomes except persistence. For example, the effects on enrollment in a four-year institution 
were positive and significant for 3 of the 11 pairs of network and non-network schools included 
in the study and not significant for the others. A significant negative effect was observed in one 
school pair. The effects were positive but not significant for four school pairs and negative but not 
significant for three school pairs. (See the Technical Appendix, section V.E.) Similarly, the estimated 
effects on enrolling in a selective institution were positive and significant for 3 of the 10 network 
schools included in the analysis, positive but not significant for four school pairs, and negative 
but not significant for three school pairs. 18 

There is evidence that the effects of attending a network school on postsecondary enrollment 
were stronger for students who entered with lower achievement than for those who entered 
with higher achievement. In particular, the effect of attending a network school on postsecondary 
enrollment in any type of postsecondary institution, as well as the effect on enrollment in four-year 
institutions, was significant and positive for students who entered high school with below-average 
achievement. In contrast, effects on these outcomes were not significant for students who 
entered high school with above-average achievement. (See the Technical Appendix, section V.F.) 

As discussed earlier, a primary goal of the deeper learning initiative is to improve equity — in 
other words, to offer opportunities to and improve outcomes for all students, particularly 
traditionally underserved student populations and struggling students. The results provide 
some indication that the network schools have made some progress toward this goal in terms 
of postsecondary enrollment. 19 


18 In one school pair, zero students at the network school enrolled in a selective institution. This pair was excluded from analyses 
for this postsecondary outcome. 

19 Additional analyses examined whether the effects of attending a network school on postsecondary enrollment and persistence 
differed among students who were eligible for free or reduced-price lunch (FRPL) and those who were not; and among males and 
females. We found that the effects on postsecondary enrollment were similar among FRPL-eligible students and students who 
were not eligible for free or reduced-price lunch. The effects of attending a network school differed for males and females on one 
outcome: enrollment in four-year postsecondary institutions. The effect of attending a network school was significant and positive 
for male students but was not significant for female students. (See the Technical Appendix, section V.F.) 
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Box 10: Design Limitations 


While this study used a rigorous design with strict school and student selection criteria to ensure that we had 
sufficient numbers of students within and across pairs to make valid conclusions, we note a few limitations. 
First, because this was a proof-of-concept study (rather than an evaluation of the networks’ approaches), we 
only included schools that implemented the network models to a moderate or high standard, which means 
that the findings cannot be generalized to all schools that are trying to implement approaches to deeper 
learning. Furthermore, the sample ultimately did not include some of the network schools that had implemented 
the models to the highest standard (due to the application of selection criteria relating to school size, grade 
range, or ongoing participation in other studies, for example) and it focused only on two specific state contexts 
(California and New York). 

In addition, while the network schools were not academically selective in admissions, in some cases students 
self-selected into the network schools. These students may have differed in some unmeasured ways from 
students with otherwise similar characteristics and prior performance who did not choose to attend a network 
school. Participants from network schools may also have been more invested in the study due to their interest 
in deeper learning, although we found no evidence to suggest that this was the case. Finally, despite having 
sufficient numbers of participating students and strong initial matching procedures, non-consent and 
non-response reduced the sample for some analyses. To adjust for this limitation, we took non-consent and 
non-response into account in the analysis (see the Technical Appendix, section IV.A for more details). 


Key Takeaways 

The findings outlined in this report support the study’s overarching conclusion that high schools 
focused on deeper learning — when the approach is at least moderately well implemented — can 
produce better outcomes for students from a diversity of backgrounds. While we note several 
limitations of our study design in Box 10, the consistency of these findings across multiple 
domains of student outcomes, and among a set of schools with varied approaches to fostering 
deeper learning, provides strong evidence for the following takeaways from our analysis: 

1. On average, students who attended the network schools in the study achieved higher 
scores on the OECD PISA-Based Test for Schools (PBTS) — a test that assesses core 
content knowledge and complex problem-solving skills — than did similar students who 
attended non-network high schools. Students who attended network schools scored higher 
on all three PBTS subjects tested (reading, mathematics, and science). They also earned 
higher scores on the state ELA and mathematics tests. 

2. Students who attended participating network schools reported more positive interpersonal 
and intrapersonal outcomes than students who attended non-network schools. In particular, 
they reported higher levels of collaboration skills, academic engagement, motivation to learn, 
and self-efficacy. There were no significant differences between students who attended 
network and non-network schools on reported creative thinking skills, perseverance, locus 
of control, or self-management. 

3. Students who attended participating network schools were more likely to graduate from 
high school on time (within four years of entering Grade 9) than were students who 
attended non-network high schools. The graduation rate among students who attended 
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network schools was estimated to be about 9 percentage points higher than it was among 
similar students who attended non-network schools. 

4. Students who attended participating network schools and non-network schools had 
similar rates of enrollment in postsecondary institutions overall. However, students who 
attended network schools were more likely to enroll in four-year institutions and in 
selective institutions. One potential explanation for these results is that admission to 
four-year colleges and selective institutions is more dependent on the kinds of cognitive, 
interpersonal, and intrapersonal competencies fostered by the network schools than 
admission to postsecondary institutions in general. 

5. Although there were significant positive effects of attending a network school averaging 
across the pairs of network and non-network schools in our sample, for many outcomes — 
for example, PBTS mathematics scores — the effects of attending a network school 
varied significantly across individual pairs of schools. Given the fact that different schools 
adopted different approaches to fostering deeper learning, and that non-network comparison 
schools also differed in their goals and strategies for students, this variation is not 
unexpected. However, we do not yet have specific explanations for this variation. 

6 . Attending a network school had similar benefits for students who entered high school 
with low achievement and for those who entered with high achievement, particularly for 
the test score and high school graduation outcomes. However, while attending a network 
school increased the postsecondary enrollment rate of students who entered high school 
with low achievement, it had no effect on the postsecondary enrollment rate of students 
who entered with high achievement. Nonetheless, attending a network school did more 
positively influence academic engagement, motivation to learn, perseverance, and self- 
management among high-achieving students. 

Taken together, the findings of the Study of Deeper Learning: Opportunities and Outcomes 
demonstrate that it is possible for schools to implement approaches that foster positive student 
outcomes across a variety of measures. We observed that attending a network school had positive 
effects on cognitive competencies as well as interpersonal and intrapersonal skills, high school 
graduation rates, and rates of enrollment in four-year institutions and selective institutions. 
Furthermore, our results indicate that network schools are able to achieve these outcomes both 
for students who enter with lower achievement and for students who enter with higher achievement. 

This study’s theory of action hypothesized that network schools would provide students with 
greater exposure to opportunities intended to foster deeper learning, and the results presented 
in the second report (Bitter et al . , 2014) provide support for this hypothesis. In addition, the 
results in our second report indicated that students with greater exposure to opportunities 
intended to foster deeper learning reported higher levels of interpersonal and intrapersonal 
competencies. Taken together, these results suggest that the key mechanisms underlying the 
positive effects of attending a network school reported in this study are the learning opportunities 
enacted within the schools — in particular, opportunities to engage in complex problem solving and 
creative thinking; opportunities to communicate, collaborate, learn how to learn, and receive 
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feedback; and opportunities for assessments aligned with deeper learning, interdisciplinary 
learning, and making real-world connections. 

While these results are promising, they also raise questions for further work. First, because we 
chose schools that were implementing networks’ approaches to fostering deeper learning at least 
moderately well, we do not know what results would have been obtained had we sampled network 
schools that implemented network approaches with lower levels of fidelity. In addition, while we 
know the schools in our sample provided opportunities for many students to engage in activities 
intended to promote deeper learning, we do not know how the schools we studied managed 
to reach and sustain their level of implementation. Further research on the implementation of 
deeper learning approaches is clearly needed to scale up the practices identified in our first 
report (Huberman et al., 2014). 

Second, our results showed substantial variation across pairs in the effects of attending network 
schools. Although we have documented this variation across schools, we do not have the evidence 
required to explain this variation. Are some ways of structuring students’ opportunities more likely 
to be effective than others? This is a key avenue for further research. 

Third, our results provide some indication that the network schools promoted equity in students' 
deeper learning outcomes. In particular, we found that the effects of attending network schools 
on the PBTS and state tests were similar for students entering the schools with low and high 
achievement, and there was evidence that low-achieving students benefited more than high- 
achieving students in terms of postsecondary enrollment. On the other hand, the results indicate 
that high-achieving students benefited more than lower-achieving students on some intrapersonal 
skills. It is not yet clear how these differences can be explained. 

Finally, while we found that attending a network school had effects on the cognitive, intrapersonal, 
and interpersonal competencies we measured, we do not know which of these competencies are 
crucial for postsecondary success. At the end of the study, study participants in the oldest cohort 
had reached only the start of their third year of postsecondary education (if they had progressed 
on time) and students in two of the five cohorts were still in high school. To draw stronger 
conclusions about the lasting effects of attending a network school, students will need to be 
followed over a longer time frame. 

The results reported within the three reports of the Study of Deeper Learning: Opportunities and 
Outcomes support deeper learning both as a focus of reform and as an agenda for further research. 
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